HUMAN NEUROSCIENCE 



ORIGINAL RESEARCH ARTICLE 

published: 02 July 2013 
doi: 10.3389/fnhum.2013.00322 



Neural stages of spoken, written, and signed word 
processing in beginning second language learners 

Matthew K. Leonard 1 - 2 ™*, Naja Ferjan Ramirez 23 *, Christina Torres 12 , Maria Hatrak 3 , 
Rachel I. Mayberry 3 * and Eric Halgren t2 ' 4 - s$ 

' Department of Radiology, University of California, San Diego, La Jolla, CA, USA 

2 Multimodal Imaging Laboratory, Department of Radiology University of California, San Diego, La Jolla, CA, USA 

3 Department of Linguistics, University of California, San Diego, La Jolla, CA, USA 

4 Department of Neurosciences, University of California, San Diego, La Jolla, CA, USA 

5 Kavli Institute for Brain and Mind, University of California, San Diego, La Jolla, CA, USA 



Edited by: 

John J. Foxe, Albert Einstein College 
of Medicine, USA 

Reviewed by: 

John S. Butler, Albert Einstein 
College of Medicine, USA 
Evie Malaia, University of Texas at 
Arlington, USA 

Correspondence: 

Matthew K. Leonard, Multimodal 
Imaging Laboratory, Department of 
Radiology, University of California, 
San Diego, 8950 Villa La Jolla Drive, 
Suite C101, La Jolla, CA 92037, USA 
e-mail: mkleonard@cogsci. ucsd. edu 
1 Present address: 
Matthew K. Leonard, Department of 
Neurological Surgery, University of 
California, San Francisco, 
San Francisco, CA, USA 

* These co-first authors have 
contributed equally to this work. 

* These co-senior authors have 
contributed equally to this work. 



We combined magnetoencephalography (MEG) and magnetic resonance imaging (MRI) 
to examine how sensory modality, language type, and language proficiency interact 
during two fundamental stages of word processing: (1) an early word encoding stage, 
and (2) a later supramodal lexico-semantic stage. Adult native English speakers who 
were learning American Sign Language (ASL) performed a semantic task for spoken 
and written English words, and ASL signs. During the early time window, written words 
evoked responses in left ventral occipitotemporal cortex, and spoken words in left 
superior temporal cortex. Signed words evoked activity in right intraparietal sulcus that 
was marginally greater than for written words. During the later time window, all three 
types of words showed significant activity in the classical left fronto-temporal language 
network, the first demonstration of such activity in individuals with so little second 
language (L2) instruction in sign. In addition, a dissociation between semantic congruity 
effects and overall MEG response magnitude for ASL responses suggested shallower 
and more effortful processing, presumably reflecting novice L2 learning. Consistent with 
previous research on non-dominant language processing in spoken languages, the L2 ASL 
learners also showed recruitment of right hemisphere and lateral occipital cortex. These 
results demonstrate that late lexico-semantic processing utilizes a common substrate, 
independent of modality, and that proficiency effects in sign language are comparable to 
those in spoken language. 

Keywords: second language acquisition, sign language, speech, reading, proficiency, modality, magneto- 
encephalography, MRI 



INTRODUCTION 

Humans acquire language in an astonishingly diverse set of 
circumstances. Nearly everyone learns a spoken language from 
birth and a majority of individuals then follow this process by 
learning to read, an extension of their spoken language experi- 
ence. In contrast to these two tightly-coupled modalities (written 
words are a visual representation of phonological forms, spe- 
cific to a given language), there exists another language form 
that bears no inherent relationship to a spoken form: Sign lan- 
guage. When deaf children are raised by deaf parents and acquire 
sign as their native language from birth, they develop profi- 
ciency within the same time frame and in a similar manner to 
that of spoken language in hearing individuals (Anderson and 
Reilly, 2002; Mayberry and Squires, 2006). This is not surprising 
given that sign languages have sublexical and syntactic complex- 
ity similar to spoken languages (Emmorey, 2002; Sandler and 
Lillo-Martin, 2006). Neural investigations of sign languages have 
also shown a close correspondence between the processing of 
signed words in deaf (Petitto et al., 2000; MacSweeney et al., 
2008; Mayberry et al., 2011; Leonard et al., 2012) and hearing 



native signers (MacSweeney et al., 2002, 2006) and spoken words 
in hearing individuals (many native signers are also fluent in a 
written language, although the neural basis of reading in deaf 
individuals is largely unknown). The predominant finding is that 
left anteroventral temporal, inferior prefrontal, and superior tem- 
poral cortex are the main loci of lexico-semantic processing in 
spoken/written (Marinkovic et al., 2003) and signed languages, as 
long as the language is learned early or to a high level of profi- 
ciency (Mayberry et al., 2011). However, it is unknown whether 
the same brain areas are used for sign language processing in 
hearing second language (L2) learners who are beginning to 
learn sign language. This is a key question for understanding the 
generalizability of L2 proficiency effects, and more broadly for 
understanding language mechanisms in the brain. 

In contrast to the processing of word meaning, which occurs 
between ~200^00ms after the word is seen or heard (Kutas 
and Federmeier, 2011), processing of the word form and sub- 
lexical structure appears to be modality-specific. Written words 
are encoded for their visual form primarily in left ventral occipi- 
totemporal areas (McCandliss et al., 2003; Vinckier et al., 2007; 
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Dehaene and Cohen, 2011; Price and Devlin, 2011). Spoken 
words are likewise encoded for their acoustic-phonetic and 
phonemic forms in left-lateralized superior temporal cortex, 
including the superior temporal gyrus/sulcus and planum tem- 
porale (Hickok and Poeppel, 2007; Price, 2010; DeWitt and 
Rauschecker, 2012; Travis et al., in press). Both of these pro- 
cesses occur within the first ~ 170 ms after the word is pre- 
sented. While an analogous form encoding stage presumably 
exists with similar timing for sign language, no such process has 
been identified. The findings from monolingual users of spo- 
ken/written and signed languages to date suggest at least two 
primary stages of word processing: An early, modality-specific 
word form encoding stage (observed for spoken/written words 
and hypothesized for sign), followed by a longer latency response 
that converges on the classical left fronto-temporal language net- 
work where meaning is extracted and integrated independent 
of the original spoken, written, or signed form (Leonard et al., 
2012). 

Much of the world's population is at least passingly famil- 
iar with more than one language, which provides a separate set 
of circumstances for learning and using words. Often, an L2 is 
acquired later with ultimately lower proficiency compared to the 
native language. Fluent, balanced speakers of two or more lan- 
guages have little difficulty producing words in the contextually 
correct language, and they understand words as rapidly and effi- 
ciently as words in their native language (Dunabeitia et al, 2010). 
However, prior to fluent understanding, the brain appears to go 
through a learning process that uses the native language as a 
scaffold, but diverges in subtle, yet important ways from native 
language processing. The extent of these differences (both behav- 
iorally and neurally) fluctuates in relation to the age at which L2 
learning begins, the proficiency level at any given moment dur- 
ing L2 learning, the amount of time spent using each language 
throughout the course of the day, and possibly the modality of the 
newly-learned language (DeKeyser and Larson-Hall, 2005; van 
Heuven and Dijkstra, 2010). Thus, L2 learning provides a unique 
opportunity to examine the role of experience in how the brain 
processes words. 

In agreement with many L2 speakers' intuitive experi- 
ences, several behavioral studies using cross-language translation 
priming have found that proficiency and language dominance 
impact the extent and direction of priming (Basnight-Brown 
and Altarriba, 2007; Dunabeitia et al., 2010; Dimitropoulou 
et al., 2011). The most common finding is that priming is 
strongest in the dominant to non-dominant direction, although 
the opposite pattern has been observed (Duyck and Warlop, 
2009). These results are consistent with models of bilin- 
gual lexical representations, including the Revised Hierarchical 
Model (Kroll and Stewart, 1994) and the Bilingual Interactive 
Activation + (BIA+) model (Dijkstra and van Heuven, 2002), 
both of which posit interactive and asymmetric connections 
between word (and sublexical) representations in both lan- 
guages. The BIA+ model is particularly relevant here, in that 
it explains the proficiency-related differences as levels of acti- 
vation of the integrated (i.e., shared) lexicon driven by the 
bottom-up input of phonological/orthographic and word-form 
representations. 



An important question is how these behavioral proficiency 
effects manifest in neural activity patterns: Does the brain pro- 
cess less proficient words differently from more familiar words? 
Extensive neuroimaging and neurophysiological evidence sup- 
ports these models, and shows a particularly strong role for profi- 
ciency in cortical organization (van Heuven and Dijkstra, 2010). 
Two recent studies that measured neural activity with magnetoen- 
cephalography (MEG) constrained by individual subject anatomy 
obtained with magnetic resonance imaging (MRI) found that, 
while both languages for Spanish-English bilinguals evoked activ- 
ity in the classical left hemisphere fronto-temporal network, 
the non-dominant language additionally recruited posterior and 
right hemisphere regions (Leonard et al., 2010, 2011). These areas 
showed significant non-dominant > dominant activity during an 
early stage of word encoding (between ~ 100-200 ms), continuing 
through the time period typically associated with lexico-semantic 
processing (~200-400 ms). Crucially, these and other studies 
(e.g., van Heuven and Dijkstra, 2010) showed that language pro- 
ficiency was the main factor in determining the recruitment of 
non-classical language areas. The order in which the languages 
were acquired did not greatly affect the activity. 

These findings are consistent with the hemodynamic imag- 
ing and electrophysiological literatures. Using functional MRI 
(fMRI), proficiency-modulated differences in activity have been 
observed (Abutalebi et al., 2001; Chee et al., 2001; Perani and 
Abutalebi, 2005), and there is evidence for greater right hemi- 
sphere activity when processing the less proficient L2 (Dehaene 
et al., 1997; Meschyan and Hernandez, 2006). While fMRI pro- 
vides spatial resolution on the order of millimeters, the hemo- 
dynamic response unfolds over the course of several seconds, 
far slower than the time course of linguistic processing in the 
brain. Electroencephalographic methods including event-related 
potentials (ERPs) are useful for elucidating the timing of activity, 
and numerous studies have found proficiency-related differ- 
ences between bilinguals' two languages. One measure of lexico- 
semantic processing, the N400 [or N400 m in MEG; (Kutas and 
Federmeier, 2011)] is delayed by ~40-50ms in the L2 (Ardal 
et al, 1990; Weber-Fox and Neville, 1996; Hahne, 2001), and 
this effect is constrained by language dominance (Moreno and 
Kutas, 2005), in agreement with the behavioral and MEG studies 
discussed above. In general, greater occipito-temporal activity in 
the non-dominant language (particularly on the right), viewed 
in light of delayed processing, suggests that lower proficiency 
involves less efficient processing that requires recruitment of 
greater neural resources. While the exact neural coding mecha- 
nism is not known, this is a well-established phenomenon that 
applies to both non-linguistic (Carpenter et al., 1999) and high- 
level language tasks (St George et al, 1999) at the neuronal 
population level. 

The research to date thus demonstrates two main findings: 
(1) In nearly all subject populations that have been examined, 
lexico-semantic processing is largely unaffected by language 
modality with respect to spoken, written, and signed lan- 
guage, and (2) lower proficiency involves the recruitment of a 
network of non-classical language regions that likewise appear 
to be modality-independent. In the present study, we sought to 
determine whether the effects of language proficiency extend to 
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hearing individuals who are learning sign language as an L2. 
Although these individuals have extensive experience with a visual 
language form (written words), their highly limited exposure 
to dynamic sign language forms allows us to investigate profi- 
ciency (English vs. ASL) and modality (spoken vs. written, vs. 
signed) effects in a single subject population. We tested a group 
of individuals with a unique set of circumstances as they relate to 
these two factors. The subjects were undergraduate students who 
were native English speakers who began learning American Sign 
Language (ASL) as an L2 in college. They had at least 40 weeks 
of experience, and were the top academic performers in their 
ASL courses and hence able to understand simple ASL signs and 
phrases. They were, however, unbalanced bilinguals with respect 
to English/ ASL proficiency. Although there have been a few previ- 
ous investigations of highly proficient, hearing L2 signers (Neville 
et al., 1997; Newman et al., 2001), no studies have investigated 
sign language processing in L2 learners with so little instruction. 
Likewise, no studies have investigated this question using meth- 
ods that afford high spatiotemporal resolution to determine both 
the cortical sources and timing of activity during specific process- 
ing stages. Similar to our previous studies on hearing bilinguals 
with two spoken languages, here we combined MEG and struc- 
tural MRI to examine neural activity in these subjects while they 
performed a semantic task in two languages/modalities: spoken 
English, visual (written) English, and visual ASL. 

While it is not possible to fully disentangle modality and 
proficiency effects within a single subject population, these fac- 
tors have been systematically varied separately in numerous 
studies with cross-language and between-group comparisons 
(Marinkovic et al, 2003; Leonard et al, 2010, 201 1, 2012), and are 
well-characterized in isolation. It is in this context that we exam- 
ined both factors in this group of L2 learners. We hypothesized 
that a comparison between the magnitudes of MEG responses 
to spoken, written, and signed words would reveal a modality- 
specific word encoding stage between ~ 100-200 ms (left superior 
planar regions for spoken words, left ventral occipitotempo- 
ral regions for written words, and an unknown set of regions 
for signed words), followed by stronger responses for ASL (the 
lower proficiency language) in a more extended network of brain 
regions used to process lexico-semantic content between ~200- 
400 ms post-stimulus onset. These areas have previously been 
identified in spoken language L2 learners and include bilateral 
posterior visual and superior temporal areas (Leonard et al, 2010, 
2011). Finding similar patterns for beginning ASL L2 learners 
would provide novel evidence that linguistic proficiency effects 
are generalizable, a particularly striking result given the vastly 
different sensory characteristics of spoken English and ASL. We 
further characterized the nature of lexico-semantic processing 
in this group by comparing the N400 effect across modalities, 
which would reveal differences in the loci of contextual inte- 
gration for relatively inexperienced learners of a visual second 
language. 

MATERIALS AND METHODS 
PARTICIPANTS 

Eleven hearing native English speakers participated in this study 
(10 F; age range = 19.74-33.16 years, mean = 22.42). All were 



healthy adults with no history of neurological or psychological 
impairment, and had normal hearing and vision (or wore correc- 
tive lenses that were applied in the MEG). All participants had 
at least four academic quarters (40 weeks) of instruction in ASL, 
having reached the highest level of instruction at either UCSD 
or Mesa College. Participants were either currently enrolled in a 
course taught in ASL or had been enrolled in such a course in the 
previous month. One participant had not taken an ASL course in 
the previous 4 months. Participants completed a self-assessment 
questionnaire that asked them to rate their ASL proficiency on a 
scale from 1 to 10. For ASL comprehension, the average score was 
7.1 ± 1.2; ASL production was 6.5 ± 1.9; Fingerspelling compre- 
hension was 6.4 ± 1.6; and fingerspelling production was 6.8 ± 
1.7. Six participants reported using ASL on a daily basis at the 
time of enrollment in the study, while the remaining participants 
indicated weekly use (one participant indicated monthly use). 

Participants gave written informed consent to participate in 
the study, and were paid $20/h for their time. This study was 
approved by the Institutional Review Board at the University of 
California, San Diego. 

STIMULI AND PROCEDURE 

In the MEG, participants performed a semantic decision task that 
involved detecting a match in meaning between a picture and a 
word. For each trial, subjects saw a photograph of an object for 
700 ms, followed by a word that either matched ("congruent") or 
mismatched ("incongruent") the picture in meaning. Participants 
were instructed to press a button when there was a match; 
response hand was counterbalanced across blocks within subjects. 
Words were presented in blocks by language/modality for spo- 
ken English, written English, and ASL. Each word appeared once 
in the congruent and once in the incongruent condition, and 
did not repeat across modalities. All words were highly image- 
able concrete nouns that were familiar to the participants in both 
languages. Since no frequency norms exist for ASL, the stimuli 
were selected from ASL developmental inventories (Schick, 1997; 
Anderson and Reilly, 2002) and picture naming data (Bates et al., 
2003; Ferjan Ramirez et al., 2013b). The ASL stimuli were piloted 
with four other subjects who had the same type of ASL instruc- 
tion to confirm that they were familiar with the words. Stimulus 
length was the following: Spoken English mean = 473.98 ± 
53.17 ms; Written English mean = 4.21 ± 0.86 letters; ASL video 
clips mean = 467.92 ± 62.88 ms. Written words appeared on the 
screen for 1500 ms. Auditory stimuli were delivered through ear- 
phones at an average amplitude of 65 dB SPL. Written and signed 
word videos subtended <5 degrees of visual angle on a screen 
in front of the subjects. For all stimulus types, the total trial 
duration varied randomly between 2600 and 2800 ms (700 ms 
picture + 1500 ms word container + 400-600 ms inter- trial 
interval). 

Each participant completed three blocks of stimuli in each lan- 
guage/modality. Each block had 100 trials (50 stimuli in each 
of the congruent and incongruent conditions) for a total of 
150 congruent and incongruent trials in each language/modality. 
The order of the languages/modalities was counterbalanced 
across participants. Prior to starting the first block in each lan- 
guage/modality, participants performed a practice run to ensure 
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they understood the stimuli and task. The practice runs were 
repeated as necessary until subjects were confident in their per- 
formance (no subjects required more than one repetition of the 
practice blocks). 

MEG RECORDING 

Participants sat in a magnetically shielded room (IMEDCO- 
AG, Switzerland) with the head in a Neuromag Vectorview 
helmet-shaped dewar containing 102 magnetometers and 204 
gradiometers (Elekta AB, Helsinki, Finland). Data were collected 
at a continuous sampling rate of 1000 Hz with minimal filtering 
(0.1 to 200 Hz). The positions of four non-magnetic coils affixed 
to the subjects' heads were digitized along with the main fidu- 
ciary points such as the nose, nasion, and preauricular points 
for subsequent coregistration with high-resolution MR images. 
The average 3-dimensional Euclidian distance for head move- 
ment from the beginning of the session to the end of the session 
was 7.38 mm (SD = 5.67 mm). 

ANATOMICALLY-CONSTRAINED MEG (aMEG) ANALYSIS 

The data were analyzed using a multimodal imaging approach 
that constrains the MEG activity to the cortical surface as deter- 
mined by high-resolution structural MRI (Dale et al., 2000). This 
noise-normalized linear inverse technique, known as dynamic 
statistical parametric mapping (dSPM) has been used extensively 
across a variety of paradigms, particularly language tasks that 
benefit from a distributed source analysis (Marinkovic et al, 2003; 
Leonard et al., 2010, 2011, 2012; Travis et al., in press), and has 
been validated by direct intracranial recordings (Halgren et al., 
1994; McDonald et al., 2010). 

The cortical surface was obtained in each participant with 
a Tl-weighted structural MRI, and was reconstructed using 
FreeSurfer. The images were collected at the UCSD Radiology 
Imaging Laboratory with a 1.5T GE Signa HDx scanner using 
an eight-channel head coil (TR = 9.8 ms, TE = 4.1ms, 77 = 
270 ms, flip angle = 8°, bandwidth = ±15.63 kHz, FOV = 24 cm, 



Left Right 




FIGURE 1 | Diagram of bilateral ROI locations. These ROIs were 
selected based on an independent data set that compared sign in native 
deaf individuals to speech in hearing individuals (Leonard et al., 2012). 
1 , Inferior Prefrontal; 2, Anterior Insula; 3, Planum Temporale; 4, Superior 
Temporal Sulcus; 5, Posterior STS; 6, Intraparietal Sulcus; 7, Lateral 
Occipito-temporal cortex; 8, Temporal Pole; 9, Inferior Temporal; 10, Ventral 
Occipito-temporal cortex. 



matrix = 192 x 192, voxel size = 1.25 x 1.25 x 1.2 mm). All Tl 
scans were collected using online prospective motion correction 
(White et al., 2010). A boundary element method forward solu- 
tion was derived from the inner skull boundary (Oostendorp and 
Van Oosterom, 1992), and the cortical surface was downsampled 
to ~2500 dipole locations per hemisphere (Dale et al, 1999; Fischl 
et al., 1999). The orientation-unconstrained MEG activity of each 
dipole was estimated every 4 ms, and the noise sensitivity at each 
location was estimated from the average pre-stimulus baseline 
from —190 to —20 ms for the localization of the subtraction for 
congruent- incongruent trials. 

The data were inspected for bad channels (channels with 
excessive noise, no signal, or unexplained artifacts), which were 
excluded from all further analyses. Additionally, trials with large 
(>3000 fT for gradiometers) transients were rejected. Blink 
artifacts were removed using independent components analysis 
(Delorme and Makeig, 2004) by pairing each MEG channel with 
the electrooculogram (EOG) channel, and rejecting the indepen- 
dent component that contained the blink. On average, fewer than 
five trials were rejected for each condition. 

Individual participant dSPMs were constructed from the aver- 
aged data in the trial epoch for each condition using only data 



Table 1 | Mean reaction time and accuracy data across 
languages/modalities. 




Language/modality 






Spoken English Written English ASL 


RT (SD) ms 582.64 (77.51 ) 542.80 (92.81 ) 
Accuracy (SD) % 97.91(1.14) 96.00(3.46) 


719.49 (92.67) 
89.36 (5.01) 


Table 2 | Summary of MEG effects by time window. 


Effect 


Time window 






Early Lexico-semantic 
encoding (congruity) 


Lexico-semantic 
(overall) 


A > W&S 


L PT R STS 

LSTS 

RSTS 




W> A&S 


L vOT 




S > A&W 


R IPS 


L IFG 
L IT 
R IFG 
R IPS 
R LOT 
R PT 
R pSTS 


A&W > S 


L IFG 
L IT 
LpSTS 
L PT 
LSTS 
R IFG 
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from the gradiometers, and then these data were combined across 
subjects by taking the mean activity at each vertex on the cor- 
tical surface and plotting it on an average brain. Vertices were 
matched across subjects by morphing the reconstructed cortical 
surfaces into a common sphere, optimally matching gyral-sulcal 
patterns and minimizing shear (Sereno et al., 1996; Fischl et al., 
1999). 

All statistical comparisons were made on region of interest 
(ROI) timecourses from these group data. ROIs were based on 
a separate data set not included in this study that compared 
signed and spoken word processing in congenitally deaf and 
hearing subjects using the same task presented here (Figure 1; 
Leonard et al., 2012). These ROIs were originally drawn on 



the grand average activity across both deaf and hearing partic- 
ipants, and thus are not biased toward either signed or spoken 
words. In the 80-120 ms time window, we specifically tested 
bilateral planum temporale (PT) and superior temporal sul- 
cus (STS) because these areas showed significant responses to 
spoken words, and are known to be involved in early word 
encoding in the auditory modality (Uusvuori et al., 2008; 
Travis et al, in press). For the 150-200 ms time window, we 
were specifically interested in ventral occipitotemporal (vOT) 
cortex because it is involved in written word form encod- 
ing (Vinckier et al., 2007). While there are no previous stud- 
ies of this stage for signed words, we selected bilateral intra- 
parietal sulcus (IPS) because it has been implicated in some 



a Congruent - Incongruent Subtraction 

Left 1 1 "= 1 1 RiQht 




b Grand Average 

Left J— t 1 4- Ri 9 h > 



:tft=t= 



^ 130 fT/cm 




L. Temporal 




'Y 400ms " 


^ 130 fT/cm 


R. Parietal 







FIGURE 2 | Individual subject waveforms showing sensor-level 
language/modality effects. (A) Congruent-lncongruent waveforms for each 
modality for a left temporal channel (top) show greater responses for auditory 
("AUD"; blue) and written ("WRT"; red) than for signed ("SIGN"; black) words. 
In a right parietal channel (bottom), there are no strong responses in any 



condition. (B) Grand average waveforms for each modality for a left temporal 
channel (top) show an early word encoding peak at ~100 ms for auditory words, 
followed by overlap between all three conditions at ~400ms. In the same 
right parietal channel (bottom), signed words evoke an early and persistent 
response that is stronger than the responses for both English modalities. 
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studies of non-temporally specific sign processing (MacSweeney 
et al., 2002; Emmorey et al., 2005), and because it showed 
the strongest activity during this time window. For the lexico- 
semantic time window from 300 to 400 ms, we tested all ten 
bilateral ROIs. With the exceptions of IPS and lateral occip- 
itotemporal (LOT) cortex, these areas are typically involved 
in lexico-semantic processing, including anteroventral tempo- 
ral areas that are hypothesized to be largely supramodal. We 
also included LOT because it has been implicated as a lexico- 
semantic area that is modulated by proficiency (Leonard et al., 
2010, 2011). 

RESULTS 

REACTION TIME AND ACCURACY 

The following analyses use these abbreviations: A = auditory 
words, W = written words, and S = signed words. Participants 
performed within ranges similar to those of native speakers and 
signers on the semantic decision task in both reaction time and 
accuracy compared to results in other studies (Leonard et al., 
2012; Travis et al., in press). Table 1 shows the average and stan- 
dard deviations for each language/modality. A one-way AN OVA 
comparing reaction times across modalities revealed a signifi- 
cant effect of modality [Fn 30) = 12.21, p < 0.0002]. Consistent 
with the fact that English was the subjects' native and dom- 
inant language and ASL was a recently learned L2, reaction 
times were significantly faster for A than for S [t(\o) = 6.85, 
p < 0.0001], and for W than for S [t (m = 8.22, p < 0.0001]. 
A and W were not significantly different. Similarly, there was a 
significant effect of modality for accuracy on the semantic task 
[F(2, 30) = 17.31, p < 0.00001]. Participants were more accurate 
for A than for S [t(io) = 5.13, p < 0.0001], and for W than for S 
[t(io) = 4.13, p = 0.002], although accuracy for S was still quite 
good (nearly 90%). Accuracy for A and W were not significantly 
different. 

aMEG RESULTS SUMMARY 

There were distinct patterns of neural activity related to the lan- 
guage/modality subjects saw or heard and language proficiency. 
These effects began during the earliest stages of word encod- 
ing (~ 100 ms for auditory, and ~ 150 ms for written and signed 
words), and continued through lexico-semantic encoding (~300- 
400 ms). Table 2 summarizes the main findings by time window, 
and the following sections statistically describe the effects shown 
in the table and figures. Figure 2 shows sensor-level data from a 
single representative subject. 

aMEG— 80-120 MS (EARLY WORD ENCODING) 

Previous investigations have identified an evoked potential peak 
at ~ 100 ms that shows selectivity for auditory speech stimuli 
compared to sensory controls in primarily left superior tempo- 
ral and superior planar cortex (Uusvuori et al., 2008; Travis et al., 
in press). We tested the MEG response during this window in two 
areas to determine whether an auditory-selective modality effect 
was present. We found a main effect of modality in left planum 
temporale (PT) 10 ) = 3.58, p = 0.047], and in left superior 
temporal sulcus (STS) [F (h 10) =6.22, p = 0.008] (Figure 3). 
The effect in PT was driven by trending A>W [t(io) = 1-95, 



p = 0.079] and A>S [r (10) = 1.93, p = 0.083] responses, and 
likewise for STS [t (m = 2.77, p = 0.02; t m = 2.37, p = 0.039] 
(Figure 4). Similar effects were obtained in right PT [i 7 ^ 10) = 
6.15, p = 0.008] and STS [F (h m = 10.74, p = 0.001]. While 
the right STS effect was driven by an A>W [f(io) = 4.00, p = 
0.003] and A>S [f(io) = 2.81, p = 0.018] response, the right PT 
effect showed an overall smaller response to W compared with A 
[f(io) = 3.32, p = 0.008] and S [f(i 0 ) = 3.00, p = 0.013]. Thus, 
during the 80-120 ms time window, the brain showed a preferen- 
tial response for auditory words primarily in superior temporal 
areas. 

aMEG— 150-200 MS (EARLY WORD ENCODING) 

The early word encoding response to written words occurs later 
than for auditory words, and is centered in a left posterior ventral 
occipitotemporal (vOT) region. During a window from 150 to 
200 ms, we tested for a W>A and W>S effect in vOT (Figure 5). 
In the left hemisphere, there was a main effect of modality 
[F (h 10) = 4.57, p = 0.023], driven by W>A [f (10 ) = 4.58, p = 
0.001] and W>S [f ( i 0) = 2.36, p = 0.04] responses (Figure 4). 
The homologous right hemisphere vOT region did not show 
significant effects (ps > 0.5). 

Given that there are early word encoding processes for auditory 
and written words, it is reasonable to ask whether such a pro- 
cess exists for signed words. We examined the response to signs 
from 150 to 200 ms, when we expect post-sensory, but pre-lexical 
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FIGURE 3 | Grand average group dSPMs during the early encoding 
time window from 80 to 120ms. (A) Auditory words ("AUD") show 
strong responses in bilateral PT and STS. (B) Written ("WRT") and (C) 
signed ("SIGN") words show sensory processing at the occipital pole. 
F-values on the color bars represent signal-to-noise ratios. 
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FIGURE 4 | ROI timecourses for the grand average across each 
language/modality. At 80-1 20 ms in left PT, auditory words (blue lines) 
show a strong modality-specific peak. From 150 to 200 ms, written words 
(red lines) show a word encoding peak in left vOX and signed words 
(black lines) show a word encoding effect in right IPS. During a later 
time window from 300 to 400 ms (thick gray bars), all conditions show 



similar responses in most left hemisphere regions, but signed words 
show much stronger responses in right hemisphere regions, including 
LOT, IPS, and PT. Asterisks represent statistically significant differences. 
Abbreviations: IPS, intraparietal sulcus; LOT, lateral occipitotemporal; PT, 
planum temporale; STS, superior temporal sulcus; TP temporal pole; vOT, 
ventral occipitotemporal. 



processing to occur. The dSPM timecourses in Figure 4 revealed 
a S>A and S>W pattern in right intraparietal sulcus (IPS), and 
indeed this region showed a marginal main effect of modal- 
ity [F(i : io) = 3.20, p = 0.062]. Post-hoc tests revealed a signifi- 
cant S>W response [f(io) = 2.51, p = 0.031], but the differences 
between W & A and S & A were not significant (Figure 5). 

aMEG— 300-400 MS (LEXICO-SEMANTIC PROCESSING) 

Based on results from a previous study that compared sign 
processing in deaf native signers to spoken word processing in 
hearing native English speakers using the same task presented 
here (Leonard et al., 2012), and on previous work examining both 
early and late processing of spoken vs. written words (Marinkovic 
et al., 2003), we selected ten ROIs to investigate sensitivity to 
semantic congruity: Inferior prefrontal cortex, anterior insula, 
planum temporale, superior temporal sulcus, posterior supe- 
rior temporal sulcus, intraparietal sulcus, lateral occipitotemporal 
cortex, temporal pole, inferior temporal cortex, and ventral occip- 
itotemporal cortex (Leonard et al., 2012). For each language and 
modality, A, W, and S, we calculated dSPMs of the subtraction 
of incongruent-congruent words, extracted timecourses for each 
subtraction condition, and tested for within-subject effects of lan- 
guage and modality. Since this procedure isolates brain activity 
evoked by incongruent vs. congruent trials, it follows that any sig- 
nificant activity indicates the localization of N400-like semantic 
congruity effects. 

We calculated an omnibus ANOVA with three within-subject 
factors: Language/modality (3), ROI (10), and hemisphere (2). 



There were highly significant main effects of language/modality 
[F( 2 , 20) = 6.96, p = 0.005], ROI [F (9 , 90) = 6.76, p < 0.0001], 
and hemisphere [Fci i io) = 10.07, p = 0.01]. There were sig- 
nificant interactions between language/modality and ROI 
[F(is, 180) = 2.35, p = 0.002], and language/modality and 
hemisphere [-F(2, 20) = 9.75, p = 0.001], but no three-way 
interaction. 

Based on a priori hypotheses about specific ROIs from pre- 
vious studies (see Materials and Methods), we tested a series 
of planned comparisons across modalities. Overall, there was 
a highly similar response to A, W, and S words (Figure 6). 
While A and W showed semantic effects of a similar magni- 
tude, these were weaker for S across most regions (Figure 7). 
In the left hemisphere, there was a main effect in infe- 



rior frontal cortex [F( 



1, 10) = 9.92, p = 0.001], driven by A>S 
[£(10) = 3.81, p = 0.003] and W>S [f (10 ) = 3.29, p = 0.008] 
responses. Similarly, in inferior temporal (IT) cortex, there was 
an effect of modality [F (u m = 5.94, p = 0.009] with A>S 
[t ( io) = 2.40, p = 0.038] and W>S [f ( i 0) = 3.50, p = 0.006]. 
In posterior STS (pSTS), there was a significant difference 
[F(i 10 ) = 4.97, p = 0.018], driven primarily by a W>S response 
[f ( i 0) =3.09, p = 0.011] and a trend for A>S [f ( i 0) = 1.98, 
p = 0.075]. Superior temporal regions showed main effects of 
modality where all three conditions differed significantly from 
one another [PT: F ih m = 15.03, p < 0.0001; STS: F(i, i 0) = 
24.71, p < 0.0001]. None of the other five left hemisphere 
ROIs showed significant differences between language/modality 
effects. 
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FIGURE 5 | Grand average group dSPMs during the early encoding 
time window from 150 to 200 ms. (A) Auditory words ("AUD") continue 
to evoke activity in bilateral superior temporal cortex, while (B) Written 
words ("WRT") show a modality-specific peak in left vOT. (C) Signed words 
("SIGN") show a modality-specific peak in right IPS. F-values on the color 
bars represent signal-to-noise ratios. 
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FIGURE 6 | Congruent-lncongruent subtraction dSPMs during the late 
lexico-semantic time window from 300 to 400 ms. (A-C) All three 
conditions show similar patterns of activity in predominantly left 
fronto-temporal regions, including PT, STS, inferior frontal, and anteroventral 
temporal. (C) Signed words ("SIGN") show overall smaller subtraction 
effects. F-values on the color bar represent signal-to-noise ratios. 



In the right hemisphere, S elicited smaller responses in 
inferior frontal cortex i 0 ) = 4.70, p = 0.021], with W>S 
[f(io) = 2.66, p = 0.024] and a marginal A>S difference [t(\o) = 
2.14, p = 0.058]. In STS, there was a main effect of modal- 
ity [.F(i, io) = 5.68, p = 0.011], driven primarily by a strong 
A>S response [f(io) = 3.51, p = 0.006] and a trend for A>W 
[f (10 ) = 1.88, p= 0.09]. None of the other eight right hemi- 
sphere ROIs showed significant language/modality effects. Thus, 
lexico-semantic congruity effects occurred in similar areas across 
languages/modalities, but with a smaller magnitude for signed 
words. 

aMEG— 300-400 MS (OVERALL RESPONSES) 

To understand which regions responded to words in each lan- 
guage/modality, but which were not necessarily influenced by 



semantic context, we also examined the grand average responses 
of congruent and incongruent trials together at 300-400 ms. 
While the previous analysis demonstrated small congruity effects 
for signed words, examination of the grand average revealed a dif- 
ferent pattern (Figure 8). In the same ROIs, we tested these grand 
averages for language/modality effects (Figure 4). 

In the left hemisphere, inferior frontal cortex showed a main 
effect of language/modality [Fq 10 ) = 3.65, p = 0.044] with 
S>W [f (10) = 2.36, p = 0.04] and S>A [t (10) = 2.76, p = 0.02]. 
IT showed a similar marginal effect io) = 3.35, p = 0.056], 
driven by a marginal S>W effect [t(io) = 2.21, p = 0.052] and 
a trend for S>A [f (10) = 2.05, p = 0.067]. None of the other 
eight left hemisphere ROIs showed significant language/modality 
effects. 
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FIGURE 7 | ROI timecourses for the Congruent-lncongruent subtraction 
across each language/modality. From 300 to 400 ms (thick gray bars), 
auditory (blue lines) and written (red lines) words evoke stronger effects than 



signed words (black lines). This difference is most prominent in the classical 
left fronto-temporal language network. Asterisks represent statistically 
significant differences. 



In the right hemisphere, we observed widespread effects where 
signs evoked greater activity than auditory or written words. 
Inferior frontal cortex showed this pattern [F^ 10 ) = 10.78, p = 
0.001] withS>W [t(i 0 ) = 3.19,/) = 0.01] andS>A[f (10) = 3.85, 
p = 0.003]. The same pattern was found for IPS [F ih 10) = 19.81, 
p < 0.0001] with S>W [f a o) = 7.03, p < 0.0001] and S>A 
[£(10) = 3.85, p = 0.003]. In lateral occipitotemporal (LOT) cor- 
tex, there was a main effect of language/modality [F^ io) = 6.21, 
p = 0.008] withS>W [f(i 0 ) = 2.89, p = 0.016] andS>A [f ( i 0) = 
2.62, p = 0.026]. Similarly, language/modality effects were appar- 
ent in PT io) = 5.09, p = 0.016] with S>W [t m = 2.76, 
p = 0.02] and S>A [t m = 2.44, p= 0.035]) and in pSTS 
10) = 4.97, p = 0.018] with S>W ]f ( io) = 3.38, p = 0.007] 
and S>A [t (w) = 2.01, p = 0.072]). The other five right hemi- 
sphere ROIs did not show significant language/modality effects. 
To summarize, although all languages/modalities showed simi- 
lar lexico-semantic congruity effects in the classical left fronto- 
temporal language network, the overall response magnitude to 
signed words was greater primarily in right hemisphere regions. 

It is possible that the overall aMEG responses contain a 
bias when looking at between-modality differences if the effects 
are not of similar magnitudes for both congruent and incon- 
gruent trials. Therefore, we conducted an additional analysis 
that compared S, W, and A words for congruent and incon- 
gruent trials separately (Figure 9). The one-way ANOVAs for 
each ROI showed that there were significant differences for con- 
gruent trials in right IPS [Fq 30) = 5.33, p = 0.01] and right 
LOT [F(2, 30) = 5.68, p = 0.008]. For incongruent trials, the 
same pattern was significant for the following right hemisphere 
ROIs: IPS [F (2 , 30) = 20.07, p < 0.0001], LOT [F (2 , 30) = 6.36, 

p = 0.0004], PT [F (2 , 30) = 
= 5.116, p = 0.01]. Thus, the 



p = 0.005], IFG [F (2 , 30) = 10.37, 
4.84, p = 0.015], and pSTS 30) 



right hemisphere effects we observed with the combined con- 
gruent/incongruent grand average are observed consistently in 
analyses with only incongruent trials, and also for only congruent 
trials in two ROIs. 

DISCUSSION 

In the present study we examined the spatiotemporal dynam- 
ics of word processing across spoken and written English and 
ASL in a group of hearing, English native speakers who were 
beginning L2 learners of ASL. During an early word encoding 
stage (~ 100 ms for spoken English, and ~ 150 ms for written 
English and ASL), words evoked activity in modality-specific 
brain regions. Responses to English words in the auditory and 
visual modalities conformed to previous findings in superior 
temporal and ventral occipitotemporal areas, respectively. ASL 
signs evoked a strong response in right IPS, although the activ- 
ity was only marginally significantly larger than for written and 
spoken English. During a later time window associated with 
lexico-semantic processing, a distributed network of bilateral 
regions responded to a semantic congruity manipulation. Several 
classical left fronto-temporal language areas showed stronger 
modulation for English (the native language) in spoken and 
written modalities relative to the L2, ASL. However, when we 
examined the overall activity during this time window, by dis- 
regarding congruity effects, signed words evoked greater activity 
than both spoken and written words in a network of mostly 
right hemisphere regions. See Table 2 for a summary of the 
results. 

The early modality-specific word encoding responses are con- 
sistent with a large number of previous studies using a variety 
of methodologies. For written words, we observed a peak in left 
vOT, a region that has been shown to be important for reading, 
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FIGURE 8 | Grand average group dSPMs during the late 
lexico-semantic time window from 300 to 400 ms. (A-C) All three 
conditions show a similar pattern of activity in bilateral regions. (C) Signed 
words ("SIGN") show much stronger activity, particularly in the right 
hemisphere. F-values on the color bars represent signal-to-noise ratios. 



and specifically for constructing written word-forms (McCandliss 
et al., 2003; Vinckier et al, 2007; Dehaene and Cohen, 201 1; Price 
and Devlin, 2011). Although there is evidence that it is a multi- 
modal region (Price and Devlin, 2003), it does seem to play an 
important role in encoding written words. In addition to the loca- 
tion, the peak timing of the activity in this region at ~ 170 ms 
is consistent with previous electrophysiological and neuroimag- 
ing studies (McCandliss et al, 2003; McDonald et al, 2010). 
Additionally, although written and signed words are perceived 
through the visual modality, signs did not evoke activity in this 
region in this group of beginning L2 learners of ASL. It is there- 
fore possible that early encoding activity in left vOT is specific to 
static written word forms. 

Also consistent with previous studies, we observed that 
areas typically associated with encoding spoken words include 



a bilateral network of superior temporal and superior planar 
regions (Hickok and Poeppel, 2007; Price, 2010). Many of these 
areas are sensitive to subtle sublexical and phonetic manipu- 
lations (Uusvuori et al., 2008) including the presence of the 
fundamental frequency (Parviainen et al., 2005) and alterations 
in voice-onset time (Frye et al., 2007). Specific neural popula- 
tions within superior temporal cortex have been found to encode 
categorical and phoneme-selective information within the first 
~150ms (Chang et al., 2010; Travis et al., in press). While the 
mechanisms and specific representations in superior temporal 
areas are unknown, research suggests that between ~60-150ms, 
the brain encodes spoken word information at a sublexical level. 
The timing and location of the peak for spoken words in the 
present study is consistent with the majority of this previous 
work. 

To date, there have not been any investigations into an anal- 
ogous stage for sign encoding. In part, this may be due to the 
fact that most previous studies have used hemodynamic methods 
that do not afford sufficient temporal resolution to distinguish 
between early and late processing stages. During a time window 
analogous to the well-established encoding processes for written 
and spoken words, ASL signs showed an activity peak in right IPS, 
which was only marginally stronger than for English words. It is 
unclear whether such activity reflects linguistic encoding (anal- 
ogous to sublexical amplitude envelope information in spoken 
language, for example) or quasi-gestural sensory characteristics 
related to space and motion (Decety and Grezes, 1999; Grossman 
and Blake, 2002; Malaia et al, 2012). The early right IPS activ- 
ity has multiple possible interpretations, and may not be related 
to the fact that the stimuli were signed words, but rather to the 
proficiency of the participants in ASL. While prior studies have 
not found right IPS to be modulated by language proficiency, the 
participants in those studies have typically possessed higher pro- 
ficiency in L2 (Leonard et al., 2010, 2011). In case studies of deaf 
signers with scant proficiency in any language, we have observed 
right IPS activation later at 300-350 ms (Ferjan Ramirez et al., 
2013a). It is possible that modality and proficiency interact in 
parietal regions, perhaps reflecting a neural processing strategy 
that is uniquely useful for the dynamic visual linguistic content of 
sign languages. To fully disentangle these effects, and to unam- 
biguously identify the analogous word encoding stage for sign 
languages, it will be necessary to conduct studies with native deaf 
and hearing signers and low proficiency deaf and hearing sign- 
ers using carefully controlled stimuli that separate linguistic and 
sensory levels of processing [similar to recent work with spoken 
words (Travis et al., in press)]. These experiments are yet to be 
carried out, and the present results provide both anatomical and 
functional brain regions that can be used to test the interaction 
between proficiency and modality. 

The results for word meaning and higher-level language 
encoding processes were more definitive and demonstrated that 
proficiency effects translate across spoken, written, and signed 
words. Beginning at ~200ms, all three word types were pro- 
cessed in a highly similar left-lateralized network including 
inferior frontal, superior temporal, and anteroventral tempo- 
ral areas. These regions have been hypothesized to provide core 
support for lexico-semantic encoding at a supramodal level 
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FIGURE 9 | Mean dSPM values for all 10 right hemisphere ROIs, 
analyzed separately for congruent and incongruent trials. (A) 

One-Way ANOVAs testing for differences across modalities on congruent 



trials were significant in IPS and LOT. (B) Effects for incongruent trials 
were significant in IFG, IPS, LOT, PT, and pSTS. *p < 0.05; **p<0.01; 
"*p < 0.001. 



(Marinkovic et al., 2003; Lambon Ralph et al., 2010), and are 
the main neural generators of the N400 response (Halgren et al., 
1994; Marinkovic et al., 2003), even in infants who are only 
beginning to acquire language (Travis et al., 2011). These areas 
all showed semantic modulation in the congruent/incongruent 
picture-word matching task (albeit to a lesser extent for ASL). 
Analyses of the overall magnitude of the response to words in 
each language and modality showed that spoken, written, and 
signed words all evoke strong activity in these regions, consistent 
with previous intracranial recordings showing locally-generated 
differential responses to different semantic categories of words in 
the anterior temporal lobe, regardless of modality (Chan et al, 
2011). Previous hemodynamic studies have found activity related 
to lexico-semantic processing in these areas for sign language 
(Petitto et al., 2000; MacSweeney et al., 2008; Mayberry et al., 
2011), and N400 responses have been observed to sign (Neville 
et al., 1997). To our knowledge, however, this is the first demon- 
stration of such activation patterns after so little L2 instruction 
in ASL. These findings provide strong support for the hypothesis 
that these areas, especially in the anterior temporal lobe, func- 
tion as supramodal hubs for high-level semantic representations 
(Patterson et al, 2007; Visser and Lambon Ralph, 2011), and 
seem difficult to explain as reflecting either knowledge of unique 
entities or social conceptual knowledge (Simmons and Martin, 
2009). 

The different patterns we observed for the congruent- 
incongruent subtraction and the grand average of all activity 



provide a window into the nature of lexico-semantic processing. 
Up to this point, we have focused on modality-specific effects. We 
now turn to how the design of this study provides insights into 
the role of language experience on the neural processing of words. 
The participants had an average of almost 23 years of extensive 
experience with spoken English, approximately 19 years of expe- 
rience with written English, but only a few months of primarily 
classroom instruction in ASL. Proficiency has profound effects on 
neural activity, and even on brain structure (Maguire et al., 2000). 
Numerous studies have demonstrated experience-related differ- 
ences in bilingual language processing (Abutalebi et al., 2001; 
Chee et al, 2001; Perani and Abutalebi, 2005; Leonard et al., 2010, 
201 1; van Heuven and Dijkstra, 2010). These studies further show 
that a surprisingly small amount of L2 exposure is required to 
elicit automatic lexico-semantic processing (McLaughlin et al., 
2004). The present results demonstrate that this is true for begin- 
ning L2 learning of ASL as well. 

As would be expected, an examination of the lexico-semantic 
effects in the present study indicates that proficiency-modulated 
activity also occurs in sign processing. In particular, we found that 
ASL words evoked greater grand average activity than both spo- 
ken and written English in a network of mostly right hemisphere 
regions (the two left hemisphere regions that were significant in 
the grand average, IFG and IT, were not significant when congru- 
ent and incongruent trials were analyzed separately). It is striking 
that some of these areas (right LOT, pSTS, and IFG) are nearly 
identical to those that showed a non-dominant > dominant 
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pattern in hearing Spanish-English bilinguals (Leonard et al., 
2010, 2011). The results for L2 ASL learners provide additional 
evidence that these areas play an important role in processing 
words in a less proficient language. The present results, together 
with our previous findings, demonstrate that word processing 
in a less proficient L2 shows increased activity in these regions 
(particularly for semantically incongruent words) relative to word 
processing in the native language. The recruitment of these areas 
for both spoken and sign language L2 processing indicates that 
they function as an additional supramodal resource for processing 
meaning in a non-dominant language. 

The dissociation between semantic congruity and overall 
activity across languages provides a finer-grained characteriza- 
tion of how proficiency affects neural processing. The English > 
ASL congruity effects in left fronto-temporal areas could sug- 
gest shallower or less complete processing of semantic content 
in the non-dominant language. The slower reaction times and 
lower accuracy for ASL support this hypothesis. However, given 
that subjects performed the task relatively well indicates that 
some neural processing strategy was used successfully. The ASL > 



English responses in the grand average MEG activity across 
both hemispheres suggest that additional neural resources were 
recruited to perform the task, although perhaps not at the same 
semantic depth. The overall stronger ASL > English differences 
for incongruent words compared to congruent words support this 
hypothesis. As these L2 learners improve their ASL proficiency, we 
predict that the grand average activity will decrease to English- 
like levels, and the congruent/incongruent difference will likewise 
increase. This represents a testable hypothesis for tracking neural 
processing strategies during development (Schlaggar et al., 2002; 
Brown et al., 2005) and later language acquisition in a bilingual 
context. 
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